As an important component of the surface meteorological observation, the daily observation of surface frost still relies on manual labor. Therefore, a new method for detecting frost based on computer vision was proposed. First, a k-nearest neighbor graph model was constructed by incorporating the manually labeled frosty image samples and the test samples which were acquired during the real-time detection. Second, the candidate frosty regions were extracted by rating those test samples using a graph-based manifold learning procedure which took the aforementioned frosty samples as the query nodes. Finally, those candidate frosty regions were identified by an on-line trained classifier based on Support Vector Machine (SVM). Some experiments were conducted in a standardized weather station and the manual observation was taken as the baseline. The experimental results demonstrate that the proposed method achieves an accuracy of 87% in frost detection and has a potential applicability in the operational surface observation.